Heuristics for Faster Error Detection With Automated Black Box Testing

Three building blocks for test guidance algorithms, the step evaluation, the state evaluation and the evaluation order, are proposed in this paper. We show how a simple family of coverage criteria can be used to evaluate individual testing steps, and how the nondeterministic behaviour of the tested system can be handled and longer term test step plans created with the state evaluation. We use the evaluation order to define which and when states are evaluated. Six heuristic algorithms based on these ideas are implemented. Four of them use a game-like approach to black box testing. In addition, three other test guidance algorithms are implemented for comparison. The algorithms are compared by measuring the number of testing steps required for detecting errors that are infiltrated to the conference protocol systems of two different sizes.